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All claims being allowable, PROSECUTION ON THE MERITS IS (OR REMAINS) CLOSED in this application. If not included 
herewith (or previously mailed), a Notice of Allowance (PTOL-85) or other appropriate communication will be mailed in due course. THIS 
NOTICE OF ALLOWABILITY IS NOT A GRANT OF PATENT RIGHTS. This application is subject to withdrawal from issue at the initiative 
of the Office or upon petition by the applicant. See 37 CFR 1.313 and MPEP 1308. 
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requirement and election have been incorporated into this action. 

3. ^ The allowed claim(s) is/are 10-16 . 

4. ^ Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
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1 . ^ Certified copies of the priority documents have been received. 

2. □ Certified copies of the priority documents have been received in Application No. . 

3. □ Copies of the certified copies of the priority documents have been received in this national stage application from the 

International Bureau (PCT Rule 17.2(a)). 
* Certified copies not received: . 

Applicant has THREE MONTHS FROM THE "MAILING DATE" of this communication to file a reply complying with the requirements 
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Paper No./Mail Date . 
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DETAILED ACTION 
Continued Examination Under 37 CFR 1.114 

1 . A request for continued examination under 37 CFR 1.114, including the fee set forth in 
37 CFR 1.17(e), was filed in this application after final rejection. Since this application is 
eligible for continued examination under 37 CFR 1.1 14, and the fee set forth in 37 CFR 1.17(e) 
has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 
37 CFR 1.114. Applicant's submission filed on 08 March 2010 has been entered. 

2. Claims 10, 15 and 16 are currently amended, claims 11-14 are as previously presented 
and claims 1-9 were previously cancelled, thus, claims 10-16 are pending in the application. 

EXAMINER'S AMENDMENT 

3. An examiner's amendment to the record appears below. Should the changes and/or 
additions be unacceptable to applicant, an amendment may be filed as provided by 37 CFR 
1.312. To ensure consideration of such an amendment, it MUST be submitted no later than the 
payment of the issue fee. 

4. Authorization for this examiner's amendment was given in a telephone interview with 
Lawrence Stahl, # 30,110, on 01 September 2011. 



5. Claim 16 has been changed to add the term non- transitory after the first word in the claim 
and is as follows: 
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Claim 16, (Currently Amended) A non-transitory computer-readable storage medium encoded 
with a computer program for an image processing method of compositing an image of a virtual 
object and an image of a physical space to generate a mixed reality image and causing a HMD to 
display the mixed reality image, comprising the steps of: holding 3D CG rendering data and 
scene graph data, wherein the 3D CG rendering data contains data representing geographical 
shape and color of the virtual object, texture data, and data representing the position and 
orientation of the virtual object, and wherein the scene graph data is data in which the 3D CG 
rendering data is hierarchical managed on the basis of the parent-child relationship between 
components of the virtual object; providing an image capturing unit, which is attached to the 
HMD, to capture the image of the physical space; measuring a position and orientation of the 
HMD with a first measurement unit; operating an object manipulation unit, by a user wearing the 
HMD, in order to position and orient the virtual object; measuring a position and orientation of 
the object manipulation unit with a second measurement unit; receiving a user instruction of 
editing the virtual object, input by the user using an operation panel, wherein the operation panel 
is arranged at a position in the physical space within a viewing field of the image capturing unit 
attached to the HMD and displays an operation panel image which is used for editing the virtual 
object, and contains a region for hierarchically displaying information about each of the 
components in accordance with the scene graph data held in the database; generating the 
operation panel image by using the data held in the database with an operation panel image 
generation unit and outputting the generated operation panel image to the operation panel 
arranged at the position in the physical space within the viewing field of said image capturing 
unit; updating the data held in the database according to the user instruction received via the 
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operation panel and the measurement result of the second measurement unit, and rendering, by 
using the updated data, the image of the virtual object according to the measurement results of 
the first and second measurement units; compositing the rendered image of the virtual object and 
the captured image of the physical space to generate the mixed reality image; and displaying the 
generated mixed reality image on the HMD, to which the image capturing unit is attached. 

Reasons for Allowance 

6. The following is the Examiner's statement for reasons for allowance: In response to the 
RCE and amendment received 08 March 20 1 0, the Examiner has determined that the amended 
claims have overcome all the art of record and that art deemed available in an updated search as 
shown in the record, thus claims 10-16 are in condition for allowance. 

7. With regard to claim 10, in summary claiming an image processing apparatus for 
compositing an composting an image of a virtual object and an image of a physical space to 
generate a mixed reality image and causing a HMD to display the mixed reality image, 
comprising: a database which holds 3D CG data and scene graph data, wherein the 3D CG 
rendering data contains data representing geographical shape and color of the virtual object, 
texture data, and data representing the position and orientation of the virtual object, and wherein 
the scene graph data is data in which the 3D CG rendering data is hierarchically managed on the 
basis of the parent-child relationship between components of the virtual object; an image 
capturing unit which is attached to the HMD and captures the image of the physical space; a first 
measurement unit which measures a position and orientation of the HMD; an object 
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manipulation unit which is used by a user wearing the HMD in order to operate a position and 
orientation of the virtual object; a second measurement unit which measures a position and 
orientation of said object manipulation unit; an operation panel which can be operated by the 
user, arranged at a position in the physical space within a viewing field of said image capturing 
unit attached to the HMD, displays an operation panel image which is used for editing the virtual 
object and contains a region for hierarchically displaying information about each of the 
components in accordance with the scene graph data held in the database, and is capable of 
receiving a user instruction of editing the virtual object input by the user; an operation panel 
image generation unit which generates the operation panel image by using the data held in said 
database, and outputs the generated operation panel image to said operation panel; a rendering 
unit which updates the data held in said database according to the user instruction received via 
said operation panel and the measurement result of said second measurement unit, and renders, 
by using the updated data, the image of the virtual object according to the measurement results of 
said first and second measurement units; a composition unit which composites the image of the 
virtual object rendered by said rendering unit and the image of the physical space captured by 
said image capturing unit to generate the mixed reality image; and a HMD, to which said image 
capturing unit is attached, which displays the mixed reality image generated by said composition 
unit are unique features and non-obvious over the art of record. 

8. Claim 15 and 16, claiming a method and a non-transitory computer readable medium 
respectfully, have like features and scope of claim 10 and are likewise determined to be unique 
and non-obvious over the art of record. 
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9. Any comments considered necessary by applicant must be submitted no later than the 
payment of the issue fee and, to avoid processing delays, should preferably accompany the issue 
fee. Such submissions should be clearly labeled "Comments on Statement of Reasons for 
Allowance." 

Conclusion 

10. Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

/Edward Martello/ 
Examiner, Art Unit 2628 

/XIAOM. WU/ 

Supervisory Patent Examiner, Art Unit 2628 



